Creators of multimedia presentations have long recognized the dramatic impact of well-chosen music in their artistic works. Filmmakers, for example, have included musical scores that create emotions that complement and enrich what the actors are conveying as spoken words and what the cameras are conveying as visual images projected onto a screen. Few people can remember films like “Star Wars,” “The Godfather,” “Jaws,” or “Rocky” without reliving the emotions created by their musical scores. Musical scores date back to the very creation of the movie industry, when early silent films starring Charlie Chaplin primarily relied on musical accompaniments to convey the emotions and messages of different movies. Musical scores have also been used to enhance documentaries. American composer Richard Rodgers created 13 hours of original music for the 1952 television series “Victory at Sea.”
Over 38 years later, filmmaker Ken Burns used period music (along with innovative camera zooms and pans) to make 150 year old black and white photographs spring to life in the PBS TV series “The Civil War.” Films like “The Civil War” series have probably inspired millions of amateur filmmakers to add music to their own photographic slide shows over the past 20 years. Amateurs are able to do that because of easy-to-use software created during that period. For example, an amateur using Apple's IPhoto® software can create a slide show accompanied by songs selected from his or her ITunes® library with a few clicks of a mouse. Software that allows users to create videos for dissemination on Youtube®, Google+® or Facebook® presents opportunities for users to enhance those videos by adding musical selections.
With the advent of compact disc technology, the widespread development and use of the Internet, and the availability of personal MP3 players like the IPod® device, a new industry has developed to create voice recordings of textual content (both fiction and nonfiction), which are widely marketed today as “audio books.” Some audio books use limited amounts of music for introductions and conclusions or as transitions between chapters. Most audio books, however, contain only the recorded voice of the reader.
Electronic devices like Amazon's Kindle® reader or Barnes & Noble's Nook® reader, which allow one to download the textual content of books directly to the device, are rapidly transforming the way books are distributed and marketed to the public and then read by individual consumers. In a press release dated Dec. 26, 2009, Amazon reported that its sales of electronic books on December 25 of that year surpassed its sales of physical books for the first day in its history. Four months later, Apple's first IPad® tablet was sold to the public. Among other things, the IPad® tablet provides an alternative to the Kindle® reader in the market for downloading physical books to consumers. Both the Kindle® reader and the IPad® tablet provide an electronic visual display for textual content contained in existing physical books in a more convenient and efficient manner for users. The IPad® tablet and more recent multimedia devices such as Amazon's Kindle Fire® and Barnes & Noble's Nook Tablet® allow users to download multimedia content including audio books having enhanced video and audio features.
Recognizing the value of adding music to these multimedia works, there is a need for users, such as non-musicians, to have access to pre-recorded segments of music which are appropriate to the emotional impact which the user is attempting to convey. On the one hand, there is a need for users to be able to automatically classify known musical works, either acquired or composed by the user, with a representation of the emotional content, e.g., “fear,” “suspense,” “calm,” or “majesty.” In this way, music can be catalogued, e.g., stored in a database, along with one or more emotional attributes for later access. On the other hand, there is a need for users to access catalogs of music, either acquired or composed by the user, in which the emotional content of the music has been identified for easy selection, e.g., for adding to a multi-media work.
Artificial neural networks were first proposed in the 1940s. An artificial neural network comprises a series of interconnected artificial neurons that process information using a connectionist approach. Artificial neural networks are generally adaptive, being trainable based on sample data to elicit desired behaviors. Various training methods are available, e.g., backpropagation. Artificial neural networks are generally applicable to pattern classification problems.
Artificial neural networks were first simulated on computational machines in the mid 1950s. In 1958, Rossenblatt introduced the perceptron, a feedforward artificial neural network capable of performing linear classification. Backpropagation was applied as a training method to neural networks beginning in the 1970s and 1980s. Both the perceptron and the backpropagation algorithm are now well known in the art.
Various general purpose artificial neural network software are available. These software packages allow the user to specify the operating parameters of the network, including the number of neurons and their arrangement. Once a network is created, the user may train these networks through the use of training data selected by the user. The training data, applied to the neural network with the desired output values, allows the neural network to be adapted to provide desired behavior. As an example, the “Rumelhart” program provided by Michael Dawson and Vanessa Yaremchuk of the University of Alberta allows the user to configure and train a multilayer perceptron.
Although artificial neural networks provide a general purpose pattern classification tool, such networks are only capable of producing useful output when the input data is encoded. Thus, there remains a need in the art for an efficient encoding of digital audio suitable for the application of a neural network. There also remains a need for a system and method for classification of digital audio based on emotional content.